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Abstract 

This qualitative study, investigating the claims, concerns, and issues arising 
within the design stages of problem-based learning (PBL) curriculum units, was 
conducted during two masters-level classes during the summer of 1999. A hermeneutic 
dialectic discourse among veteran teachers (who were novice PBL curriculum designers) 
was facilitated by the instructor of the course. The participating teachers evinced an 
interest in implementing problem-based learning, but indicated a desire for support 
during the construction of the actual units. The study was undertaken to uncover 
impediments to quality in the PBL curriculum design process. Findings from the study 
data include the “quality” criteria incorporated into the rubrics for evaluating each design 
product (the meet the problem document, a curriculum alignment description, a sample 
know/need to know board, a sample problem statement, an anticipated problem map, a 
list of coaching questions, a resource list, a comparison of alternative solutions, a 
capstone performance description, an assessment instrument for the capstone 
performance, and a debriefing instrument) and in the final Critiquing Guide designed by 
the stakeholders. Other conclusions within the study include such teacher perceptions as: 
confidence that teachers who had no previous training in PBL could create quality 
curriculum units; several design products had overlapping quality criteria; students in 
the class gained knowledge of rubric design; and these students were able to articulate 
reasons for implementing rubric design in their own classrooms. 
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QUALITY PROBLEM-BASED LEARNING EXPERIENCES FOR STUDENTS: 
DESIGN DELIBERATIONS AMONG TEACHERS FROM DIVERSE DISCIPLINES 



This qualitative study, investigating the claims, concerns, and issues arising 
within the design stages of problem-based learning (PBL) curriculum units, was 
conducted during two masters-level classes during the summer of 1999. A hermeneutic 
dialectic discourse among veteran teachers (who were novice PBL curriculum designers) 
was facilitated by the instructor of the course. The 17 participating teachers evinced an 
interest in implementing 

problem-based learning, but indicated a desire for support during the construction of the 
actual units. The study was undertaken to uncover impediments to quality in the PBL 
curriculum design process. 

A recent literature review yielded a wealth of material on problem-based learning. 
Further perusal, however, elicited the conclusion that current research is focusing on 
issues other than the design quality of the PBL experience. The three primary issues 
within the literature appear to be: 

♦ Effects of PBL implementation on students, 

♦ Descriptions of the utilization of PBL in various disciplines, and 

♦ “How-to” monographs describing the process of PBL. 

The following table showing selected research study titles within the recent literature 
document these themes: 



Effects on Students 


Utilization in a Discipline 


“How to” (PBL process) 


Problem-Based Learning: 
Effects on the Early 
Acquisition of Cognitive 
Skill in Medicine (Hmelo, 
1998) 


Problem-based Learning in 
Analytical Science 
Undergraduate Teaching 
(Shelton & Smith, 1998) 


A Point System for 
Management of an 
Integrated Problem-based 
Learning Curriculum 
(Rundquist, Bergdahl, 
Sandstrom, & Hammar, 
1998) 


Investigating Student 
Interactions within a 
Problem-Based Learning 
Environment in Biology 
(Guerrera & Lajoie, 1998) 


Problem-Based Learning in 
Counselor Education 
(Stewart, 1998) 


Problems as Possibilities: 
Problem-Based Learning 
for K-12 Education (Torp & 
Sage, 1998) 



The intent of this study, then, was to add to a body of knowledge not currently 
being expanded via research: that body of knowledge concerning the design of quality 
problem-based learning experiences. 



Methodology 
Participants and Setting 

A focus group of 17 teachers interested in implementing problem-based learning 
(PBL) was recruited for two sections of a masters-level class on PBL during the 1999 
summer session at North Carolina State University. As stated previously, the participants 
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in the study were veteran teachers, with experiences ranging from two to ten years, but 
novice PBL curriculum designers. The instructor of the course had received one week of 
training in the problem-based learning process and had subsequently implemented PBL 
as the sole method of instruction in an entire year-long high school biochemistry class. 
The university course was developed by this instructor, based upon lessons learned from 
this experience. 

Within the PBL college course, class activities included not only constructing a 
PBL curriculum unit, but on setting design criteria for each piece and on critiquing the 
completed product. Therefore, students within the classes created rubrics (scoring 
guides) forjudging each portion of the unit, from the meet-the-problem document to the 
debriefing session. The culminating activity of the course was the design of a critiquing 
instrument to use in assessing the quality of an entire PBL curriculum unit. 

Two texts were utilized for this course. The first, Problems as Possibilities by 
Torp and Sage (1998, p. 3) explored the “possibilities of problem-based learning as a 
natural integrating focus for relevant curriculum and meaningful student learning.” This 
text had a practical focus for teachers, showing how to implement PBL in the K-12 
classroom. The second text, Implementing Problem Based Learning in Leadership 
Development by Bridges and Hallinger (1995) provided insight into the theory and 
process of PBL. Students in the class were required to read and critically review chapters 
from both texts. 



Theoretical Framework and Design of the Study 

This study was conceived within the constructivist epistemology, which 
emphasizes the need for students to interact socially in meaning-making activities. 

Tobin, Kahle, and Fraser (1990, p. 6) state this as, “Within the constructivist view, 
learning is defined as the acquisition of knowledge by individuals through a process of 
construction that occurs as sensory data are given meaning in terms of prior knowledge.” 
Von Glaserfeld (1991) elaborates on this point and states that knowledge is always the 
result of a constructive activity and therefore cannot be transferred to a passive receiver. 
Therefore, the activities utilized within this research study (and in the PBL course) were 
framed with this view of learning (a social process in which students must actively 
participate in the airing and discussion of perceptions before these perceptions can be 
ratified, modified, or altered). 

The research performed in this study was qualitative in nature. Qualitative 
research stresses “the socially constructed nature of reality, the intimate relationship 
between the researcher and what is studied, and the situational constraints that shape 
inquiry” (Denzin & Lincoln, 1994, p. 4). Within the “socially constructed reality” of a 
masters-level classroom, the instructor collected anecdotal records and compiled 
contextual evidence in order to preserve the interplay of stakeholder interactions and 
classroom events. Such records ensure a “thick” data collection which can “provide as 
complete a data base as humanly possible in order to facilitate transferability judgments 
on the part of others who may wish to apply the study to their own situations” (Guba & 
Lincoln, 1989, p. 241). Within the PBL classroom discourse, a hermeneutic, dialectic 
process as described by Guba and Lincoln (1989) was utilized. Such a process brings to 
light the “claims, concerns, and issues” (Guba & Lincoln, 1989, p. 50) of the stakeholders 
involved in the investigation. In such a hermeneutic process, stakeholders are identified 
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and then interviewed, surveyed, sampled for opinions, asked to respond, etc. An 
important characteristic of this process is that data collection and data analysis are 
intertwined. As data are collected, they are immediately analyzed and then used to direct 
new data collection efforts. In this manner, the research relies heavily on the developing 
claims, concerns, and issues of the stakeholders, making the design of the study 
emergent. 



Data Sources, Data Collection, and Data Analysis 
Since investigation of interactions within the focus (stakeholder) groups was 
emphasized in this study, any classroom artifact which could reveal these interactions 
was important. Conversational interchanges among the stakeholders were captured on 
audio-tapes. Transcripts of these tapes provide a verbatim record of such conversations. 
These conversations reveal the thoughts of the novice PBL curriculum writers as they 
struggled to design the PBL unit and as they discussed quality criteria for each design 
product. Anecdotal records made by the primary investigator (the instructor) during and 
subsequent to class meetings also helped to capture exchanges and events. Such 
anecdotal records provide a window into the primary investigator’s on-the-spot thinking 
while teaching the class. They reveal the original source of inferences and assertions 
later shared and checked with stakeholders. 

The rubrics and the Critiquing Guide created by the teacher-stakeholders 
constitute another data source, as do the reflective journal entries compiled from these 
stakeholders. Additionally, “talkback” sheets were utilized at the end of the course to 
solicit stakeholder responses to specific questions concerning course topics. These four 
sources in particular show the criteria considered for each design product, capture 
decisions made about the relative importance of each criterion, and record the perceived 
impediments to quality construction of the PBL units. A comparison of the rubrics 
which were originally constructed for each design product and the criteria present on the 
Critiquing Guide constructed at the end of the class demonstrate the growth in 
sophistication of the stakeholders. The reflective journal entries and the responses of the 
participants to the “talkback” sheet questions helped the primary investigator both 
formulate and check assertions. 

Evaluation results and the instruments of evaluation for the course must also be 
included in the list of evidence for findings. The instruments of evaluation were, of 
course, the rubrics created by the stakeholders for each design product. Evaluation 
results were given both orally and in writing and included both peer- and instructor- 
assessment of design work. In addition, the instructor kept records of the number of 
times each design product was submitted, as products were returned for further work if 
all criteria on the stakeholder-designed rubric for the product were not addressed. 

From all these sources, the primary investigator formulated assertions and then 
utilized the hermeneutic dialectic process to check these assertions by sharing them with 
the stakeholders. In this manner, throughout the study, data collection, analysis, and 
interpretation were continuous and cyclic. 

The Process of Problem-Based Learning 
In brief, the steps in the PBL process begin with a problem. This problem is 
presented to students in the “meet-the-problem” documentation. The problem becomes 
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more defined as students separate known facts about the problem topic from unknown 
issues and capture their thoughts on a “know/need to know” board. A problem statement 
or research question is then written. Data collection to illuminate unknowns begins and 
group analysis of these data are now incorporated into the process. After several cycles 
of data collection and analysis, possible solutions to the problem are formulated. The 
potential solutions are examined in the light of all the evidence collected and the most 
viable solution is then selected. The PBL experience culminates with the public sharing 
of the solution and some type of evaluation. This evaluation may be formal or informal; 
self-, peer-, or instructor-assessed; written or oral. 




z~- 

Firsf Thoughts: Original Rubrics for Each Design Product 
By the end of this PBL masters-level course, students were eventually required to 
create eleven design products for their PBL curriculum units. These eleven products with 
brief descriptions of each were: 

1 . Meet the problem document — For the purposes of this class, the masters-level 
students decided the format for their first problem would be a written document of 
some sort (rather than as a video, role-playing exercise, visual representation, etc.). 

2. Curriculum alignment description — This piece contains a brief description of the 
targeted audience (age, typical attributes, and developmental level of the students 
who will work on the problem). 

3. Sample know/need to know board — Here, teachers create a list of “knowns” and a list 
of “unknowns” in order to anticipate the lists students will make. This exercise helps 
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the teacher define her expectations and plan coaching questions to elicit inclusion of 
“expected” items, if students do not spontaneously volunteer them. 

4. Sample problem statement — The sample problem statement, like the sample 
know/need to know boards help the teacher establish the intended curriculum to be 
addressed by the problem. Then, if students do not proceed along pathways the 
teacher hopes will be explored, coaching questions can be formulated to help them 
uncover such pathways. 

5. Anticipated problem map — The map helps the teacher visualize the different parts of 
the problem, and how concepts are related or how they may overlap. 

6. List of coaching questions — Coaching questions are written for the major student 
activities within the problem (meet the problem, know/need to know board, problem 
statement, research, and generation of solutions). Coaching questions are prompts for 
student thinking at cognitive, metacognitive, and epistemic levels (See Appendix A, 
Product Six for definitions of these terms). 

7. Resource list — Here, the teacher makes a list of resources available for student use in 
the local school. The teacher also determines that these resources are adequate and 
developmental ly appropriate. 

8. Comparison of two alternative solutions — Quality problems are ones which have 
more than one right answer. The teacher must believe this to be true, in order to 
prevent “leading” the students to one particular solutions. Therefore, before 
presenting a problem to students, the teacher should create at least two different 
acceptable solutions to the problem. 

9. Capstone performance description — The capstone performance is usually the one in 
which students share their solutions to the problem with the class or with outside 
evaluators. 

10. Assessment instrument for evaluating the capstone performance — The teacher must 
plan ahead to develop criteria for this capstone performance and should ensure that 
the capstone performance is authentic to the problem (the way information would be 
shared in the “real world” if students were working there, instead of in a school). 

1 1 . Debriefing instrument — This instrument is designed to encourage evaluation of 
alternative solutions proposed by different student groups. The intent is that the class 
will create “one best solution” from all the disparate ones reported by the student 
groups in their capstone performances. 

All but two of these design products were listed in the original syllabus of the course. 
The two which were added later were ones suggested by the masters-level teachers in the 
PBL class during class discussions within the course. These two design products were 
the curriculum alignment piece and the resource list. The circumstances which led to the 
inclusion of the curriculum alignment piece are covered in detail later in this paper. The 
resource list was added after one teacher returned to her school and consulted the media 
center there for information relevant to her problem. When she found very little 
information, she was disconcerted and concerned that the problem might not be “do-able” 
(as she put it), at her school. Bringing this to the attention of the class, she suggested that 
in a quality problem, the teacher would already know which resources would be available 
to students (and not have this come as a complete surprise). In this manner, if very few 
reference materials could be accessed at the school, the teacher could make arrangements 
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to import resources from another source. Therefore, the class thought it was important to 
add a resource list as another required design product. 

For each of the eleven products, student input was sought in creating 
the rubric (scoring guide) used to evaluate each product. Before class discussions about 
these rubrics, students read pertinent material from the two course texts. They also 
participated in two rubric design experiences. The first such experience involved creating 
a rubric for cleaning a house. The second experience, called “The Camping Trip,” asked 
students to evaluate student work on a math word problem. This word problem involved 
figuring out how much water campers should carry for a three-day camping trip. Ten 
samples of student work were distributed and the teachers in the class then separated the 
samples into 3 piles by the quality of the responses (“good, better, best;” “the good, the 
bad, and the ugly;” “good, fair, poor;” etc.) They were then asked to brainstorm criteria 
for a rubric to grade the student work samples. 

From their text readings, the teachers gleaned descriptions of the design products 
and were able to articulate parameters recommended in the literature for inclusion in a 
particular product.. From the rubric design experiences, the teachers in the class 
practiced articulating quality criteria for use in scoring guides. 

At this point, the teachers were ready to design their first rubric for the problem- 
based learning unit. For each design product, they first listed criteria which should be 
included. These criteria were captured as a checklist on the chalkboard. Then, students 
in the class were asked to prioritize the criteria by level of importance and to designate 
this level of importance with a point score. 

This process was used repetitively to create rubrics for all eleven design products. 
The original rubrics generated by this process for each of the eleven products can be 
found in Appendix A. These rubrics show the criteria that the students felt needed to be 
present in each product to ensure quality of the design. It is clearly impossible, within the 
scope of this paper, to capture all the interactions and discussions that occurred in the 
design of all eleven products. However, the development of the rubric for the first 
product (meet-the-problem document) will be discussed in detail. Then, the original 
rubric will be compared with the Critiquing Guide, which was developed later in the 
course. 



Rubric for the Meet the Problem Document 
When the students were asked to generate criteria for this design product, 
several descriptors were quickly proffered by the class. These included: 

• Authentic 

• Motivating 

• Real world 

• Interdisciplinary 

• Contains roles 

• Ill-structured 

Since this was the first rubric designed by the class, the instructor 
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asked students to define each of the terms more completely, in order to assure that the 
expectations were clear. For instance, the instructor asked, “How is ‘authentic’ different 
from ‘real world’?” 

John replied that “real world means the problem would actually happen, but 
authentic means that the way we present it is how it would be presented in real life.” 

Ellen expanded on this to explain that “authentic has more to do with the format of the 
presentation, but ‘real world’ means it’s believable - that people would actually be faced 
by this problem and would have to solve it.” 

At this point, the instructor stated, “It seems that these two words, ‘authentic’ and 
‘real world,’ then, are not so much the criteria we need, but are descriptors of the criteria. 
For example, Ellen used the word ‘format.’ Is ‘authentic’ a descriptor associated with 
‘format’?” This instructor-generated question fueled the following interchange: 

Alan: Yeah, I think format is the criteria and “authentic” is just one portion of the 
format, like it should be appropriate. 

Instructor: The format or the authentic? 

Alan: I mean, the format should be authentic and appropriate to the problem. 
Chris: Like, in the real world, if we would get a memo from our boss, telling us 
to work on a problem, then for us to make the meet the problem document a 
memo is OK. It would be stupid to create a video for a problem we’d normally 
get as a memo. 

The class discussion continued, until the students had decided to use “format” as a 
criterion and “authentic” and “appropriate to the problem” as descriptors of this criterion. 
Later, Carol, a language arts teacher, suggested that we should incorporate grammar into 
the format criterion, if we were truly defining excellence. Group discussion here resulted 
in the words, “uses standard writing practices” being added to the descriptors of the 
format category. 

The discussion then moved to the “real world” descriptor. There were several 
suggestions for a category name which would include this descriptor. After discarding 
such terms as “qualities of the problem” (which the class felt would be confusing, as we 
were attempting to design “quality” criteria) and “problem needs” (which also was 
thrown out as “confusing”), the class settled on “problem characteristics” as the name of 
the second category of criteria. A coaching question from the instructor, “So, what else, 
besides ‘real world’ should be included in this category?” began the long exchange 
captured here: 

Sandy: I think “ill -structured” [from the original checklist] goes there. 

Anna: Me, too. “Ill-structured” describes the problem. 

Instructor: I know that “ill-structured” is a term used in your Torp & Sage book. 
What does this mean? 

Anna: It means the problem is messy, that it’s complex, so it’s hard to know how 
to tackle it. 

Lois: Yeah. It’s all over the place and it’s hard to solve. 
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Instructor: OK, so if we want the problem to be all these things, what are some 
words we can use to describe what we want? Should we use ill-structured? Will 
people outside this class know what this means? 

Patricia: I don’t think so. I know the first time I read this, I thought “ill- 
structured” was bad. I mean, it sounds like something that’s “ill-structured” is not 
well-written, so it’s bad. 

Sandy: Me, too. I thought Torp & Sage were saying ill-structured was NOT the 
way to make the problem. I had to read it again to understand that we WANT 
“ill-structured.” 

Instructor: So, if we had trouble understanding “ill-structured,” what else can we 
use in its place? 

Lois: How about interdisciplinary? “Ill-structured” problems are messy like we 
said, which means they’re full of; they’ve got more than one subject in them. 

Alan: Yeah, interdisciplinary. A good problem won’t just be about science, or 
just about math, or just about economics, it’ll blend lots of different subject areas. 
Instructor: OK, interdisciplinary [writes this on the board]. OK, what else? 

John: I think Anna and Lois were onto something about the problem being hard 
to solve. If we write a good problem, the answer won’t be obvious. 

Carol: Obvious, I like that. The book says that the problem ought to be one that 
hasn’t been solved before. So, if it hasn’t been solved before, there’s no way it 
could be obvious. 

Instructor: Help me phrase this, then, Carol. 

Carol: OK. How about “no obvious solution path?” 

[Instructor writes this on the board] 

Marsha: That’s not enough, though. I’ve seen lots of math problems that seemed 
like they had no “obvious solution path” to me, but the teacher didn’t agree with 
my answer, anyway! [Class laughs] 

Instructor: Thanks, Marsha. That brings up another point. What about the 
answer to the problem? 

Patricia: There should be more than one. Like, it should be open-ended. 

Ellen: It’s like that video you showed us where the teacher said “There’s more 
than one way to skin a cat!” There ought to be multiple answers to the problem, 
not just one. 

Chris: Yeah, let’s put “no one right answer.” That way, we’ll set up a problem 
that we won’t be expecting a certain answer to. 

[Instructor adds this to the list on the board] 

Instructor: OK. So far, we have 3 ideas listed under the “Problem 
Characteristics” category: “real world,” “interdisciplinary,” and “no one right 
answer.” Are there others we should include? 

Carol: I don’t like “real world.” The others are more descriptive and that’s more 
a noun. Can we say “reality-based” instead? 

[Noting general nods of consensus, the instructor makes the change] 

Instructor: OK, but what else should we have? [No volunteers answer] If you are 
having trouble deciding, why don’t you get out your meet the problem document 
for the mosquito problem [a practice problem done by the entire class earlier] and 
see if we’ve captured all the characteristics you see there. 
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John: Hey, I know. It has a deadline. Don’t we need a deadline in every 
problem? 

After some further discussion, the class decided to incorporate a 
deadline into the “problem characteristics” category. The discussion then moved to the 
roles of the student. At first, the inclusion of student roles was discussed within the 
“problem characteristics” category. However, several students suggested that this was 
important enough to be a stand-alone category. The discussion of the “student roles” 
culminated with the articulation of this description: “Present, role is one of stakeholder 
who in real world would work on problem.” This was taken to mean that student roles 
had to be present in the meet the problem document, and that, like the problem 
presentation, the roles should be authentic (appropriate) to the problem. Alan put it like 
this: “Yeah, you wouldn’t see a clown in an office, but you’d probably see an 
accountant. So, if your problem happened in an office, an accountant’s OK, but not a 
clown.” 

After creating the “format,” “problem characteristics” and “student roles” 
categories of criteria and writing descriptors for each, the students were ready to 
terminate the discussion. However, a question from Sandy fueled a new round of 
interactions: 

Sandy: You know what worries me? It’s that story you [the instructor] told us 

about your radiation problem. About how the teachers thought it was a great idea 
and they were all interested in it, but when the students got the problem, they 
could have cared less. What if that happens with our problems? How can we 
know if the students will be interested or not? 

Instructor: Good point, Sandy. What do you [the class] think? 

Ellen: I remember that video you showed us of Dead Poets Society. How Robin 
William’s class was different from the other ones the boys went to on the first 
day. 

John: Yeah. All the other teachers kept them in the classrooms, but Robin 
Williams took them to the trophy case. 

Instructor: So, should we incorporate “change of location” into our rubric for the 
meet the problem document? 

Patricia: No, that might be too hard. Not everyone can change their class if they 
want. Besides, that’s not the only way to make things interesting for students. 
Instructor: What other ways can you think of, Patricia? 

Patricia: I think it wasn’t that Robin Williams changed the location; I think it was 
just that he did something different. 

Instructor: So, “different” is interesting? 

Marsha: Yeah, I think so. It’s like something you’re not expecting, so you’re 
surprised and that makes you interested. 

Anna: It’s the novelty of it — that’s what keeps you interested. 

Instructor: OK, novelty, [writes this on board] Anything else? 

John: That thing you’re not expecting. The unexpectedness of it. 

Instructor: “Unexpectedness” - is that a word? 

Sandy: How about “discrepant event?” 
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[Instructor adds this beside “novelty.” 

Instructor: Now, where am I putting these? Are these more problem 
characteristics? 

Chris: No, let’s make a separate category, to show how important this is. We 
could call it “motivational criteria.” 

John: Yeah, let’s add it there. 

Instructor: Are these enough descriptors of motivational? Are there any more we 
should add? 

The discussion continued, with some students suggesting adding “relevance to the 
real world” into the motivational category. They felt that real world problems would be 
more motivating to students. However, other students felt that this was adequately 
covered in the “problem characteristics” category, so was not needed in the 
“motivational” category. It came down to a class vote with the majority voting to put 
“relevance to the real world” into the motivational category, yet keep “reality-based” in 
the “problem characteristics” category. The discussion then continued as to whether all 
three descriptors must be present to make a “quality” meet the problem document. 
Consensus was reached by students on a suggestion that 2 of the 3 factors (novelty, 
discrepant event, and relevance) would be enough. 

At this point, the rubric for the meet the problem document looked like this on the 

board: 



Format 


Authentic style, appropriate to problem, uses standard 
writing practices. 


Problem 

Characteristics 


Reality-based, no one right answer, inter 
disciplinary, contains deadline, lacks obvious 
solution path 


Student Roles 


Present, role is one of stakeholder who in real 
world would work on problem 


Motivational 


Contains 2 or more of the following characteristics: 
discrepant event, novelty, relevance to real world 



The class ended for the day with this rubric on the board. Students 
were asked to write a Reflective Journal entry before leaving the classroom, however, 
and the instructor reviewed these entries before the next class. A comment made by Lois 
in her journal entry was brought before the students in the next class period. Lois wrote: 

Getting teachers to use PBL as a strategy in their classroom will be difficult 
because they feel pressured (because of the state assessment) to cover the 
curriculum. Teachers already view time as a hindrance when it comes to 
teaching. With ill structured problems taking weeks to complete, teachers must 
be convinced this approach leads to higher test scores. My ultimate goal is to 
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understand how to use PBL as a way to organize the curriculum that will alleviate 
anxieties with curriculum coverage. 

The instructor used Lois’ comments to introduce curriculum alignment of state standards 
to the problem into the class discussions. After several interchanges, the students decided 
that this piece was so important they should create another design product to address this 
issue rather than adding it to the meet the problem product. These discussions eventually 
led to the development of the rubric for the curriculum alignment design product found in 
Appendix A, Product Two. 

Thinking about the curriculum and what the problem could teach however, 
brought the students to add one final descriptor to the “problem characteristics” category 
of the rubric for the meet the problem document. After several iterations, the final 
wording which was added was “contains focal points for knowledge acquisition.” Anna 
explained this addition as, “A quality problem has to lead the students to seek answers. If 
it doesn’t, there’s no use in introducing it at all!” 

At this point, students were satisfied that all important quality criteria had been 
included in their rubric for the meet the problem product. The instructor then asked 
students to prioritize the criteria, and to assign point values to each, with the larger 
numbers going to the most important criteria. Here, however, the students balked. They 
felt that what they had created showed “the essential criteria,” as Sandy put it, and that 
each criterion was equally important. Therefore, in this original rubric, the students 
simply assigned point values of “one” for each descriptor. A careful study of the original 
rubrics listed in Appendix A will show that this trend continued until design Product 
Eight, the rubric for alternative solutions. Here, students first differentiated importance 
of criteria by assigning different point values and then continued to do so for all the 
remaining original rubrics of the design products. 

Second Thoughts: The Critiquing Guide for the Problem-Based Learning Curriculum 

Units 

As the class progressed, rubrics for all eleven design products were constructed. 
Then, the masters-level students used the rubrics as guides and actually created each of 
the eleven products for their own PBL curriculum units. One enticement (other than the 
grade for the course!) for teachers to finish and submit all their products was the promise 
that completed units would be published on the Internet on a problem-based learning 
web-site developed by the instructor of the course. The instructor explained that she 
hoped the web-site would grow to include curriculum units for all disciplines, at all levels 
of instruction. However, she was afraid this would be a slow process, as the PBL course 
was only scheduled to be taught every two years. The teachers within the course 
suggested soliciting units from the general population of teachers, rather than limiting 
authorship to only those teachers enrolled in the PBL class. When the instructor 
expressed doubts about the quality of such submissions, the idea of the Critiquing Guide 
was bom. The teachers within the class felt that such a Guide could be published on the 
web-site and could serve as an educative tool for anyone wishing to create a PBL unit. 
The Guide would also define the expected quality levels of submissions. 

When this subject was originally broached, the first suggestion was simply to 
publish the rubrics for each of the design products. However, this idea was soon 
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discarded for three reasons. The first reason, stated by Joyce, was that “point scores are 
really irrelevant. We’re not looking to give the teachers [who submit units] a grade. We 
just want the units to be complete and to contain the essential criteria.” The second 
reason the original rubrics were not used for the Critiquing Guide was articulated by 
Jessica. “We’ve had this class, and we’ve read at least two books on the subject of PBL. 
But, the teachers who wish to submit may be as ignorant as we were before this class! I 
don’t think our descriptions are clear enough for someone NOT in this class to 
understand the expectations.” One other point made by the students was that designers of 
PBL units who had never taken this class might not clearly perceive the delineations 
between design products. For example, it was suggested by Eva that “they might lump 
together the curriculum alignment and the problem map.” 

Considering all these parameters led students to believe that the construction of a 
Critiquing Guide was the path they would like to take, rather than using the original 
rubrics. In a class discussion, the students decided the Guide would have three levels for 
feedback purposes: “Commendable,” “Acceptable,” and “Acceptable with Changes.” 

As Candy put it, “This sounds better than ‘Commendable,’ ‘Acceptable,’ and ‘Rejected.’ 
We still get to reject any unit which doesn’t conform to our quality criteria, but we do it 
in such a way that the author COULD re-submit and get published on the web, if they 
wanted to and if they made the necessary changes.” 

Working with these categories and working from the original rubrics, the students 
broke into groups of two to three. Each group then tackled at least two design products, 
writing quality descriptors for these products at the three different levels (Commendable, 
Acceptable, and Acceptable with Changes). Care was taken to explain terms and to make 
the criteria as explicit as possible. 

Once one group had completed its task, the newly created Guide for that group’s 
product was critiqued by another group. In this process, the groups found that many 
criteria overlapped from one design product to another. For example, the group working 
on the meet the problem document had included a descriptor addressing student roles. 

The group working on the sample problem statement had included a very similar 
descriptor, as had the group working on the problem map. The following classroom 
conversation shows how the students resolved this issue: 

Laura: We’ve [pointing to the three groups] all got essentially the same 
descriptor, even though we’re working on three different products. 

Instructor: Do you see this a problem? Should we decide where the descriptor 
best fits? 

Laura: Well, we surely don’t need to repeat the same thing three times. This 
Guide is going to be long enough as it is! 

Bridgette: Why don’t we just write the descriptor once, but group the products 
together? 

Instructor: I don’t get it. Help me understand what you’re suggesting. 

Bridgette [at board] Well, we all have this part about student roles. So, over here, 
we just write [See Bridgette’s graphic below.] 
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Product 


Commendable 


Acceptable 


Acceptable with 
Changes 


Meet the problem 
Sample statement 
Problem map 


Student roles... 







This template, first proposed by Bridgette, was utilized for all the remaining 
descriptors which overlapped two or more design products. In this manner, through a 
jigsaw process (one group passing their work on to another group), the entire Critiquing 
Guide was constructed. 

As each piece of the Critiquing Guide was created, the instructor typed and 
distributed this piece to students. When all groups had contributed pieces, the Guide was 
altered and refined through a group discussion process. Students then field-tested the 
refined Guide, using it to critique another student’s PBL curriculum unit. After this 
activity, the class members pronounced themselves “happy” with the Guide as written 
(See Appendix B for the completed Critiquing Guide.), but listed a few constructive 
criticisms the instructor might wish to address in future iterations. These included: 

• There is no place for an “overall” assessment of the submitted unit 

• Motivation and interest to students has been overlooked 

• “Life skills” (Appendix B, page 29) need to be defined as concepts outside the 
discipline of the class (as communication or interpersonal skills 

• Strike the word “Abundant” (Appendix B, page 28) and substitute “Sufficient” but 
add the following to the statement: “ Sufficient resources are available for students to 
successfully solve the problem and resources encourage students to investigate other 
areas of the problem .” 

• It sounds as if each problem must allow a variety of capstone performances 
(Appendix B, page 29). Make it apparent that variety can be addressed by utilizing 
multiple problems. (It may be too much to expect a teacher to create multiple grading 
rubrics for diverse capstone performances in EVERY problem. However, the teacher 
can vary the type of capstone performance required from problem to problem.) 

The instructor agreed to address these concerns before publishing the Critiquing Guide on 
the problem-based learning web-site. 



Conclusions 
Level One Findings 

Findings from this research study include the “quality” criteria incorporated into 
the original rubrics for all eleven design products. These rubrics, found in Appendix A, 
document the relative importance of traits recommended for inclusion in the eleven 

ducts by these novice PBL curriculum designers. The list of design products 
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constitutes another finding of the study. While the majority of the design products were 
suggested by the instructor, two new products were created based upon recommendations 
from individual students. Although the students in the class were new to PBL, they were 
still veteran teachers. Therefore, the newly suggested products (Curriculum Alignment 



and Resource List) arose from the prior knowledge gained by these veteran teachers from 
past teaching experiences. 

Another finding of the study involves teacher attitudes about problem-based 
learning. Even though the masters-level students first expressed feelings of nervousness 
and insecurity in designing problem-based learning curriculum units, and even though 
they found some of the products difficult to design, they still expressed confidence that 
teachers who had never taken the PBL class could create quality PBL units. A collateral 
finding, however, was that the students felt such inexperienced designers would need 
clearly written guidelines to help them incorporate the desired level of quality into their 
designs. This led to another study product, the Critiquing Guide (Appendix B) for use by 
those novice PBL designers who had received no other training or support. 

In the creation of this Critiquing Guide, the masters-level students gained a “big 
picture” view of the design process for the entire PBL curriculum unit. Previous to this 
exercise, students had worked on the unit in a piece-meal fashion, creating one design 
product at the time. However, when work began on the Critiquing Guide, the students’ 
view changed from looking at one small piece to examining the larger work, in toto. 
When this perspective changed, they found that many of the criteria they had designed 
for individual pieces of the project were applicable to several design products. In this 
manner, students in the class moved from listing particular traits necessary in individual 
products to articulating essential traits of the entire unit. Therefore, the Critiquing Guide 
devised by the class does not attempt to show what each individual design product must 
include. Rather, it sets a standard of quality for the whole curriculum unit. 

Level Two Findings 

The above findings can be clearly tracked through the data reported in this paper 
and sometimes even traced to particular quotes from individual students. However, a 
second set of findings is not so readily apparent. These findings involve the utilization of 
rubrics in the PBL class and the change in teacher attitudes, thinking, and behavior as a 
result of this rubric use. 

For some of the students, this class was their first introduction to rubric design. 
Bonnie stated in a reflective journal entry, “I had never seen or used a rubric until this 
class.” Both Lois and John had heard of rubrics, or seen their use, but neither had ever 
participated in designing one. Lois stated this as, “I’ve never been asked to help create a 
rubric in which my work was to be evaluated.” John made the following observations of 
the rubric design process: 

I was truly impressed with the method in which the class determined 
the rubric for our Meet the Problem document. The instructor did 
something I have rarely seen done on any academic level: allow the 
class to decide the method for scoring the assignment. I recognize 
the seasoned teacher in our instructor, as a set of predetermined guidelines 
were on display as we worked on our own rubric. And there was also 
some very subtle, yet effective “steering” going on. On the whole, 
however, the class decided what aspects of the document are to be graded 
and the point values of those parts. 
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Even those students who had previous experiences in rubric design, however, 
gained new insights in to the importance of rubrics within the PBL curriculum unit. 
Bonnie summed this up for the class as: 

With each day, the need and intent of rubrics becomes more apparent. Today’s 
discussion on rubric development gave me a chance to propose ideas and get class 
feedback as well as the instructor’s. It was an opportunity to use a sounding 
board for some of my ideas I would like to use in my assignments and class 
exercises.. .The combination of creating our own rubrics and to turn around and 
apply the information made some concepts a lot easier. 

Thus, Bonnie saw personal benefits from the rubric design process. Joyce echoed 
Bonnie’s comments, stating “I am so glad to have an objective tool, such as the rubric, to 
use. It seems like a lot of time to write out student expected outcomes, but I think it will 
help the students to know how they will be graded and what the expectations are.” 

Jessica, like Joyce, related the use of rubrics to her own students and expanded 
on this topic with: “Teachers need to provide good feedback and give the students rubrics 
to show what they are looking for... It is only fair that students get rubrics before doing 
the work. It is not fair for teachers to make up rubrics after the work is complete.” John 
and Laura reiterated the importance of using rubrics to convey teacher expectations to 
students, and Laura reminded the class that “it is important to remember that the product I 
receive will only be as good as the rubric that I provide for my students.” 

So, while several of the masters-level students emphasized the importance of 
rubric utilization, few thought rubrics were easy to design. Again and again in reflective 
journals, this theme was reiterated and the following comments appeared: 

Laura : Writing a rubric is a very difficult process because it is hard to know what 
to include and how detailed to be. 

Anna : Writing detailed rubrics is difficulty as it forces me to examine exactly 
what I am interested I concerning the final product. 

Jessica : Today we went over howto make a rubric. It was very hard. It is 
difficult to identify what you are looking for in a project. You must do this before 
you can start making a rubric. Then you have to decide how important each item 
that you are looking for is and give it a number value. I never realized how hard 
that could be. 

As these comments demonstrate, the novice designers found it difficult to clearly define 
their expectations. Anna summarized this sentiment as “Describing quality is like 
describing the color green.” 

Although the students thought the process of rubric design was difficult, most (11 
out of 17) made some reference to implementing, continuing, or revising the use of 
rubrics in their own classrooms. Sandy decided, “In the future, I’ll make a point of 
designing our rubrics together so they meet our needs and are comprehensible to the 
kids.” Comprehension was also important to Anna who remarked, “Having felt first hand 
the relief of being graded by a comprehensible rubric when faced by an unknown teacher, 
I will endeavor to re-create that relief for my students.” 
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From these additional data, then, it is clear that within this masters-level class 

• Some of the students had limited or no prior knowledge of rubric utilization 

• Many students articulated the importance of using rubrics, citing such reasons as 
quality, fairness to students, and aligning curriculum standards to assessment 

• A majority of the students planned revision of their assessment methods, based upon 
their newly gained knowledge of rubrics. 

Educational Significance of the Study 

As problem-based learning gains in popularity and is implemented in classrooms 
throughout the nation, attention must turn to creating quality PBL experiences for 
students. This study utilized a diverse focus group, recruiting stakeholders from academic 
and vocational disciplines, to set criteria forjudging PBL curriculum units. Within this 
process, knowledge essential to development of a quality product was identified and 
impediments to the design process were articulated. The Critiquing Guide created by the 
focus group can serve as a educative tool for other novice designers and aid these novices 
in constructing rich problem-based learning units of study for their students. 




17 



19 



REFERENCES 



Bridges, E. M. & Hallinger, P. (1995). Implementing problem based learning in 
leadership development . Eugene, OR: ERIC Clearinghouse on Educational 
Management, University of Oregon. 

Denzin, N. K. & Lincoln, Y. S. (1994). Introduction: Entering the field of 
qualitative research. In N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative 
research (pp. 1-17). Thousand Oaks, CA: Sage Publications, Inc. 

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation . Newbury 
Park, CA: Sage Publications, Inc. 

Guerrera, C. P., Lajoie, S.P. (1998). Investigating student interactions within a 
problem-based learning environment in biology . San Diego, CA: American Educational 
Research Association. (ERIC Document Reproduction Service No. ED 419 819) 

Hmelo, C. (1998). Problem-based Learning: Effects on the early acquisition of 
cognitive skill in medicine. Journal of the Learning Sciences, 7 (2), 173-208. 

Rundquist, I., Bergdahl, B., Sandstrom, I., Hammar, M. (1998). A point system 
for management of an integrated problem-based learning curriculum. Education for 
Health: Change in Training and Practice, 1 1 (1), 7-14. 

Shelton, J. B. & Smith, R. F. (1998). Problem-based learning in analytical 
science undergraduate teaching. Research in Science and Technological Education, 16 
(2), 19-29. 

Stewart, J. B. (1998). Problem-based learning in counselor education. Canadian 
Journal of Counseling, 32 (1), 37-49. 

Tobin, K., Kahle, J. B., & Fraser, B. J. (1990). Windows into science 
classrooms . Bristol, PA: The Falmer Press. 

Torp, L. & Sage, S. (1998). Problems as possibilities: Problem-based learning 
for K-12 education. Alexandria, VA: Association for Supervision and Curriculum 
Development. 

Von Glaserfeld, E. (1991). Answers to Atlanta (NARST) and Washington 
(AAAS) questions . Symposium conducted at the annual meeting of the National 
Association of Research in Science Teaching, Atlanta, GA. 




r> 

& 



0 



18 



Appendix A 

Original Rubrics for All Nine Design Products 



Product One: Rubric for Meet-the-Problem Document 
Criteria Descriptor Points 



Format 



Motivational 



Problem 

Characteristics 



Student Roles 



Authentic style, appropriate to 
problem, uses standard writing 
practices (3) 

Contains 2 or more of the following 

characteristics: discrepant event, 

novelty, relevance to real world (2) 

Reality-based, no one right answer, 
interdisciplinary, contains deadline, 
lacks obvious solution path, contains 
focal points for knowledge acquisition (6) 

Present, role is one of stakeholder 
who in real world would work on 
problem (2) 



TOTAL POINTS (13) 



Product Two: Rubric for Curriculum Alignment 



Criteria 


Descriptor 


Points 


Objectives 


3 from major area, 

3 from interdisciplinary 






area 


(6) 


Student 


5 significant learner 




Characteristics 


characteristics listed 


(5) 



( 11 ) 



TOTAL 



Product Three: Rubric for Sample Know/Need to Know Boards 



Name 


Date 




Criteria 


Descriptor 


Points 


Knows 


Need minimum of 10 


(10) 


Need to Knows 


Need minimum of 7 


(7) 



TOTAL (17) 



Product Four: Rubric for Sample Problem Statement 
Role Stated (1 pt) 

Problem Stated (lpt) 

4 Conditions Stated (4 pts) 

Total Points (6 pts) 



Product Five: Rubric for Problem Map 
Contains 1 central idea (lpt) 

Contains 4 extensions (4 pts.) 

Each extension contains 2 sub-extensions (8 pts.) 

Total Points (13 pts.) 
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Product Six: Rubric for Coaching Questions 
One Cognition (Thinking) Question To Ask Students Written for Each of the 



Following Activities: 

Meet the Problem ( 1 pt) 

Know/Need to Know Board (1 pt) 

Writing the Problem Statement (1 pt) 

Information Gathering & Sharing (Research) ( 1 pt) 

Generating Possible Solutions ( 1 pt) 

One Metacognition (Learning About Thinking) Question To Ask Students 
Written for Each of the Following Activities: 

Meet the Problem (1 pt) 

Know/Need to Know Board (1 pt) 

Writing the Problem Statement (1 pt) 

Information Gathering & Sharing (Research) ( 1 pt) 

Generating Possible Solutions (1 pt) 

One Epistemic Cognition (Nature of Knowing) Question To Ask Students 
Written for Each of the Following Activities: 

Meet the Problem ( 1 pt) 

Know/Need to Know Board (1 pt) 

Writing the Problem Statement ( 1 pt) 

Information Gathering & Sharing (Research) ( 1 pt) 

Generating Possible Solutions ( 1 pt) 

TOTAL POINTS ( 1 5 pts) 



Cognitive : Students compute, read, perceive, and comprehend information. 
According to the meet the problem document, what is your role in this 
problem? 

Metacognitive : These questions help students monitor their own thinking 
process and consider appropriate strategies. If you are having difficulty 
locating resources for the research on “conjoined twins, ” what alternative 
terms might you try? 

Epistemic : Refers to the individual’s understanding of the nature of 
problems and includes knowledge about the limits and certainty of knowing, 
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and the criteria for knowing. Is it necessary to find answers to ALL your 
Need to Know questions? 



Product Seven: Rubric for Resource List 

Number of Resources (8 required) (8) 

Use of APA Style in each listing (8) 

Use of 4 different types of source material (4) 

TOTAL POINTS (20) 

Product Eight: Rubric for Alternative Solutions 

Two possible solutions to problem 

described (20) 

4 “pro” statements listed for each 

possible solution (8) 

4 “con” statements listed for each 

possible solution (8) 

2 consequences listed for each 

possible solution (4) 

1 paragraph written, stating preferred 
solution and describing rationale 
for choosing this one over the other (10) 

(50) 



Total Points 



Product Nine: Capstone Performance Description 



Is authentic to problem ( 1 5) 

Relates to stated objectives (10) 

Promotes learning of new or refinement of old 

skills/knowledge (8) 

Addresses at least 2 types of assessment (group, 

peer, self) (6) 

States audience for performance (4) 



Describes what students will do to share 
solution to problem 

Total Points (58) 



.( 15 ) 



Product Ten: Rubric for Assessment Instrument 



Contains at least 4 criteria 




(20) 


Each criterion has two descriptors 




(8) 


First descriptor details excellence 
Second descriptor details mediocre 




( 4 ) 


(not worst) performance 




( 4 ) 


Total Points 


(36) 
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Product" Eleven: Rubric for Debriefing Session 
Review of Student-Generated Solutions 

Description of the procedure for disseminating solutions is included 
and the procedure ensures that all student solutions will be shared with all 
students. lOpts. 

A description is present, but no mention of all solutions being shared 
with all students. 5 pts. 

Rating of Solutions 

A clear description of the procedure for rating all solutions is present 
(description gives examples of procedure, as using pro and con graphic 
organizers). The procedure for sharing the ratings is also clearly described. 

20 pts. 



Procedures for rating and procedures for sharing ratings are described, 
but lack of clarity impedes complete understanding. 



15 pts. 



One Best Solution 

Description of a process which leads students to formulate one “best” 
solution from all student-generated solutions and from any new issues which 
arose during the discussion of these is present. 30 pts. 

Coaching by Teacher 

Explanation of debriefing process explains how the teacher will coach 
students through this process in order to address important concepts/issues 
that did not arise during investigation of the problem. 

20 pts. 



Total Points Possible = 80 
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Appendix B 

Critiquing Guide for the Problem Based Learning Curriculum Unit 
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University of Maryland 
1129 Shriver Laboratory 
College Park, MD 20742-5701 




Clearinghouse on Assessment and Evaluat 



March 2000 
Dear AERA Presenter, 

Congratulations on being a presenter at AERA. The ERIC Clearinghouse on Assessment and 
Evaluation would like you to contribute to ERIC by providing us with a written copy of your 
presentation. Submitting your paper to ERIC ensures a wider audience by making it available to 
members of the education community who could not attend your session or this year's conference. 

Abstracts of papers accepted by ERIC appear in Resources in Education (R1E) and are announced to over 
5,000 organizations. The inclusion of your work makes it readily available to other researchers, provides a 
permanent archive, and enhances the quality of R1E. Abstracts of your contribution will be accessible 
through the printed, electronic, and internet versions of R1E. The paper will be available full-text, on 
demand through the ERIC Document Reproduction Service and through the microfiche collections 
housed at libraries around the world. 

We are gathering all the papers from the AERA Conference. We will route your paper to the 
appropriate clearinghouse and you will be notified if your paper meets ERIC's criteria. Documents 
are reviewed for contribution to education, timeliness, relevance, methodology, effectiveness of 
presentation, and reproduction quality. You can track our processing of your paper at 
http://ericae.net. 

To disseminate your work through ERIC, you need to sign the reproduction release form on the 
back of this letter and include it with two copies of your paper. You can drop of the copies of 
your paper and reproduction release form at the ERIC booth (223) or mail to our attention at the 
address below. If you have not submitted your 1999 Conference paper please send today or 
drop it off at the booth with a Reproduction Release Form. Please feel free to copy the form 
for future or additional submissions. 

Mail to: AERA 2000/ERIC Acquisitions 

The University of Maryland 
1 1 29 Shriver Lab 
College Park, MD 20742 



Sincerely, 




Lawrence M. Rudner, Ph.D. 
Director, ERIC/AE 



Tel: (800) 464-3742 
(301) 405-7449 
FAX: (301) 405-8134 
ericae@ericae.net 
http://ericae.net 



i ERIC/AE is a project of the Department of Measurement, Statistics and Evaluation 

I at the College of Education, University of Maryland. 




